Method and apparatus for generating personalized 3D face model

ABSTRACT

A method of generating a three-dimensional (3D) face model includes extracting feature points of a face from input images comprising a first face image and a second face image; deforming a generic 3D face model to a personalized 3D face model based on the feature points; projecting the personalized 3D face model to each of the first face image and the second face image; and refining the personalized 3D face model based on a difference in texture patterns between the first face image to which the personalized 3D face model is projected and the second face image to which the personalized 3D face model is projected.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of and claims priority under 35U.S.C. §§120/121 to U.S. patent application Ser. No. 14/882,624, filedon Oct. 14, 2015, which claims priority under 35 U.S.C. §119 to KoreanPatent Application No. 10-2014-0165506, filed on Nov. 25, 2014, in theKorean Intellectual Property Office, the entire contents of each ofwhich are incorporated herein by reference in their entirety.

BACKGROUND 1. Field

Example embodiments relate to technology for generating athree-dimensional (3D) face model based on an input image.

2. Description of the Related Art

When representing a face pose, an illumination, a facial expression, andthe like, three-dimensional (3D) face modeling may provide a moreprecise and realistic perception compared to two-dimensional (2D) facemodeling, and is desired in fields such as face recognition, games, andavatars. A 3D face modeling method includes a method of modeling a 3Dface of a user using a 3D depth camera or a 3D scanner. The method usinga 3D depth camera models a 3D face of a user using a stereo matchingscheme. The method using a 3D scanner models a 3D face of a user using3D data scanned through the 3D scanner. A 3D coordinate value and acolor value are acquired with respect to each of sampling points on thesurface of the face of the user, and 3D face modeling is performed onthe user face based on the acquired 3D coordinate value and color value.

SUMMARY

According to at least one example embodiment, a method of generating athree-dimensional (3D) face model includes extracting feature points ofa face from input images comprising a first face image and a second faceimage; deforming a generic 3D face model to a personalized 3D face modelbased on the feature points; projecting the personalized 3D face modelto each of the first face image and the second face image; and refiningthe personalized 3D face model based on a difference in texture patternsbetween the first face image to which the personalized 3D face model isprojected and the second face image to which the personalized 3D facemodel is projected.

The refining may include extracting a correspondence point between thefirst face image to which the personalized 3D face model is projectedand the second face image to which the personalized 3D face model isprojected; and generating a comparison based on a texture pattern of thefirst face image and a texture pattern of the second face image in aperipheral area of the correspondence point.

The refining may include refining a shape of the personalized 3D facemodel to make a similarity between the texture pattern of the first faceimage and the texture pattern of the second face image exceed athreshold degree of similarity in the peripheral area of thecorrespondence point.

The refining may include includes, iteratively, determining if a firstcondition is satisfied, and refining a shape of the personalized 3D facemodel based on the difference in texture patterns, until the firstcondition is satisfied.

The refining may include determining a pose deformation parameter and ashape control parameter that reduce the difference in texture patternsbetween the first face image and the second face image in the peripheralarea of the correspondence point; and applying the determined posedeformation parameter and shape control parameter to the personalized 3Dface model.

The shape control parameter may control a spatial location of each ofvertices constituting the personalized 3D face model.

The projecting may include projecting the personalized 3D face model tothe first face image based on a facial pose included in the first faceimage; and projecting the personalized 3D face model to the second faceimage based on a facial pose included in the second face image.

The deforming may include acquiring the personalized 3D face model bymapping landmark points of the generic 3D face model to the extractedfeature points.

The input images may be two-dimensional (2D) images, and the inputimages may include at least one frontal image captured from a front of auser face and at least one side face image captured from a side of theuser face.

According to at least one example embodiment, a non-transitorycomputer-readable medium stores computer-readable instructions that,when executed by one or more processors, cause the one or moreprocessors to perform operations including, extracting feature points ofa face from input images comprising a first face image and a second faceimage, deforming a generic 3D face model to a personalized 3D face modelbased on the feature points, projecting the personalized 3D face modelto each of the first face image and the second face image, and refiningthe personalized 3D face model based on a difference in texture patternsbetween the first face image to which the personalized 3D face model isprojected and the second face image to which the personalized 3D facemodel is projected.

According to at least one example embodiment, an apparatus forgenerating a three-dimensional (3D) face model includes a memory storingcomputer-readable instructions; and one or more processors configured toexecute the computer-readable instruction to, extract feature points ofa face from input images comprising a first face image and a second faceimage, deform a generic 3D face model to a personalized 3D face modelbased on the feature points, and refine the personalized 3D face modelbased on a difference in texture patterns between the first face imageto which the personalized 3D face model is projected and the second faceimage to which the personalized 3D face model is projected.

The one or more processors may be configured to extract a correspondencepoint between the first face image to which the personalized 3D facemodel is projected and the second face image to which the personalized3D face model is projected, and compare a texture pattern of the firstface image and a texture pattern of the second face image in aperipheral area of the correspondence point.

The one or more processors may be configured to refine a shape of thepersonalized 3D face model to make a similarity between the texturepattern of the first face image and the texture pattern of the secondface image exceed a threshold degree of similarity in the peripheralarea of the correspondence point.

The one or more processors may be configured to, iteratively, determineif a first condition is satisfied, and refine a shape of thepersonalized 3D face model based on the difference in texture patterns,until the first condition is satisfied.

The one or more processors may be configured to, determine a posedeformation parameter and a shape control parameter that reduce thedifference in texture patterns between the first face image and thesecond face image in the peripheral area of the correspondence point,and apply the determined pose deformation parameter and shape controlparameter to the personalized 3D face model.

The one or more processors may be further configured to execute thecomputer-readable instructions to store the refined personalized 3D facemodel.

The one or more processors may be configured to acquire the personalized3D face model by mapping landmark points of the generic 3D face model tothe extracted feature points.

The input images may be two-dimensional (2D) images, and the inputimages may include at least one frontal image captured from a front of auser face and at least one side face image captured from a side of theuser face.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features and advantages of example embodiments ofthe inventive concepts will become more apparent by describing in detailexample embodiments of the inventive concepts with reference to theattached drawings. The accompanying drawings are intended to depictexample embodiments of the inventive concepts and should not beinterpreted to limit the intended scope of the claims. The accompanyingdrawings are not to be considered as drawn to scale unless explicitlynoted.

FIG. 1 is a diagram to describe the overall operation of an apparatusfor generating a personalized three-dimensional (3D) face modelaccording to example embodiments;

FIG. 2 is a block diagram illustrating an apparatus for generating apersonalized 3D face model according to example embodiments;

FIG. 3 illustrates face images to describe a process of extractingfeature points from input images according to example embodiments;

FIG. 4 illustrates face models to describe a process of generating apersonalized 3D face model by deforming a generic 3D face modelaccording to example embodiments;

FIG. 5 illustrates face images to describe a process of projecting apersonalized 3D face model to input images, comparing texture patternsof the input images, and refining the personalized 3D face modelaccording to example embodiments;

FIG. 6 is a flowchart illustrating a method of generating a personalized3D face model according to example embodiments;

FIG. 7 is a flowchart illustrating in detail an operation of refining apersonalized 3D face model in a method of generating the personalized 3Dface model according to example embodiments.

DETAILED DESCRIPTION

Detailed example embodiments of the inventive concepts are disclosedherein. However, specific structural and functional details disclosedherein are merely representative for purposes of describing exampleembodiments of the inventive concepts. Example embodiments of theinventive concepts may, however, be embodied in many alternate forms andshould not be construed as limited to only the embodiments set forthherein.

Accordingly, while example embodiments of the inventive concepts arecapable of various modifications and alternative forms, embodimentsthereof are shown by way of example in the drawings and will herein bedescribed in detail. It should be understood, however, that there is nointent to limit example embodiments of the inventive concepts to theparticular forms disclosed, but to the contrary, example embodiments ofthe inventive concepts are to cover all modifications, equivalents, andalternatives falling within the scope of example embodiments of theinventive concepts. Like numbers refer to like elements throughout thedescription of the figures.

It will be understood that, although the terms first, second, etc. maybe used herein to describe various elements, these elements should notbe limited by these terms. These terms are only used to distinguish oneelement from another. For example, a first element could be termed asecond element, and, similarly, a second element could be termed a firstelement, without departing from the scope of example embodiments of theinventive concepts. As used herein, the term “and/or” includes any andall combinations of one or more of the associated listed items.

It will be understood that when an element is referred to as being“connected” or “coupled” to another element, it may be directlyconnected or coupled to the other element or intervening elements may bepresent. In contrast, when an element is referred to as being “directlyconnected” or “directly coupled” to another element, there are nointervening elements present. Other words used to describe therelationship between elements should be interpreted in a like fashion(e.g., “between” versus “directly between”, “adjacent” versus “directlyadjacent”, etc.).

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of exampleembodiments of the inventive concepts. As used herein, the singularforms “a”, “an” and “the” are intended to include the plural forms aswell, unless the context clearly indicates otherwise. It will be furtherunderstood that the terms “comprises”, “comprising,”, “includes” and/or“including”, when used herein, specify the presence of stated features,integers, steps, operations, elements, and/or components, but do notpreclude the presence or addition of one or more other features,integers, steps, operations, elements, components, and/or groupsthereof.

It should also be noted that in some alternative implementations, thefunctions/acts noted may occur out of the order noted in the figures.For example, two figures shown in succession may in fact be executedsubstantially concurrently or may sometimes be executed in the reverseorder, depending upon the functionality/acts involved.

Example embodiments of the inventive concepts are described herein withreference to schematic illustrations of idealized embodiments (andintermediate structures) of the inventive concepts. As such, variationsfrom the shapes of the illustrations as a result, for example, ofmanufacturing techniques and/or tolerances, are to be expected. Thus,example embodiments of the inventive concepts should not be construed aslimited to the particular shapes of regions illustrated herein but areto include deviations in shapes that result, for example, frommanufacturing.

Although corresponding plan views and/or perspective views of somecross-sectional view(s) may not be shown, the cross-sectional view(s) ofdevice structures illustrated herein provide support for a plurality ofdevice structures that extend along two different directions as would beillustrated in a plan view, and/or in three different directions aswould be illustrated in a perspective view. The two different directionsmay or may not be orthogonal to each other. The three differentdirections may include a third direction that may be orthogonal to thetwo different directions. The plurality of device structures may beintegrated in a same electronic device. For example, when a devicestructure (e.g., a memory cell structure or a transistor structure) isillustrated in a cross-sectional view, an electronic device may includea plurality of the device structures (e.g., memory cell structures ortransistor structures), as would be illustrated by a plan view of theelectronic device. The plurality of device structures may be arranged inan array and/or in a two-dimensional pattern.

FIG. 1 is a diagram to describe the overall operation of an apparatus100 for generating a personalized three-dimensional (3D) face modelaccording to example embodiments.

Referring to FIG. 1, according to at least some example embodiments, thepersonalized 3D face model generating apparatus 100 generates apersonalized 3D face model based on a plurality of input images. Inputimages may include input images captured from a user face to be modeledat different angles or in different directions. For example, face imagesmay include at least one frontal face image captured from a front of theuser face and at least one side face image captured from a side of theuser face. A face image refers to a two-dimensional (2D) image capturedfrom the user face, and may include a global area or a partial area ofthe user face.

According to at least one example embodiment of the inventive concepts,the 3D face model generating apparatus 100 may include or be implementedby one or more circuits or circuitry (e.g., hardware) specificallystructured to carry out some or all of the operations described hereinas being performed by the 3D face model generating apparatus 100 (or anelement thereof). According to at least one example embodiment of theinventive concepts, the 3D face model generating apparatus 100 mayinclude or be implemented by a memory and one or more processorsexecuting computer-readable code (e.g., software) that is stored in thememory and includes instructions corresponding to some or all of theoperations described herein as being performed by the 3D face modelgenerating apparatus 100 (or an element thereof). According to at leastone example embodiment of the inventive concepts, the 3D face modelgenerating apparatus 100 may be implemented by, for example, acombination of the above-referenced hardware and processors executingcomputer-readable code.

The term ‘processor’, as used herein, may refer to, for example, ahardware-implemented data processing device having circuitry that isphysically structured to execute desired operations including, forexample, operations represented as code and/or instructions included ina program. Examples of the above-referenced hardware-implemented dataprocessing device include, but are not limited to, a microprocessor, acentral processing unit (CPU), a processor core, a multi-core processor;a multiprocessor, an application-specific integrated circuit (ASIC), anda field programmable gate array (FPGA). Processors executing programcode are programmed processors, and thus, are special-purpose computers.

To generate a 3D face model representing a shape of a user face, theuser may take a plurality of 2D face images using a camera. The taken 2Dface images may be input to the personalized 3D face model generatingapparatus 100. Face images in which the user face is taken in differentdirections may be acquired by taking a photo while fixing the user faceand changing a location of the camera, or by taking a photo while fixingthe camera and changing a direction of the user face.

The personalized 3D face model generating apparatus 100 may generate apersonalized 3D face model by three-dimensionally modeling the user facebased on the input face images. The personalized 3D face modelgenerating apparatus 100 may generate the personalized 3D face model byselecting one of generic 3D face models stored in a generic 3D facemodel database 110 and by matching a shape of the selected generic 3Dface model to a shape of the user face represented in each of the faceimages. A generic 3D face model refers to a statistical 3D shape modelgenerated based on training data.

The personalized 3D face model generating apparatus 100 may generate thepersonalized 3D face model by extracting feature points of a face fromthe face images and by matching the extracted feature points to thegeneric 3D face model. Here, the “feature point of a face” or “facialfeature point” may be a “facial keypoint” or a “facial landmark”. Forexample, the personalized 3D face model generating apparatus 100 maygenerate a personalized 3D face model by extracting, from each of faceimages, feature points corresponding to eyebrows, eyes, a nose, lips, achin, ears, and/or the face contour, and by deforming a generic 3D facemodel to be matched to the extracted feature points.

The personalized 3D face model generating apparatus 100 may generate ahigh-precision personalized 3D face model by quickly generating a coarsepersonalized 3D face model based on feature points extracted from faceimages and by refining a shape of the personalized 3D face model basedon texture information of the face images.

The personalized 3D face model generating apparatus 100 may project thepersonalized 3D face model to each of the face images and may determinea correspondence point between the face images using the personalized 3Dface model. The personalized 3D face model generating apparatus 100 maycompare texture patterns of the face images in a peripheral area of thecorrespondence point, and may refine a shape of the personalized 3D facemodel, so that a similarity between the texture patterns of the faceimage may increase.

For example, a difference in texture patterns may be present in acorrespondence area between a frontal face image to which thepersonalized 3D face model is projected and a side face image to whichthe personalized 3D face model is projected. A texture corresponding toa nose end of the personalized 3D face model may represent a texture ofa nose in the frontal face image, and may represent a texture of abackground in the side face image. The personalized 3D face modelgenerating apparatus 100 may refine a shape of the personalized 3D facemodel so that the difference in texture patterns (i.e., between thefrontal face image and the side face image) may decrease.

For example, the personalized 3D face model generating apparatus 100 mayreduce a height of the nose of the personalized 3D face model until atexture of a nose end area is changed from the texture of the backgroundto the texture of the nose in the side face image to which thepersonalized 3D face model is projected. The personalized 3D face modelgenerating apparatus 100 may compare texture patterns of face images inanother correspondence area based on the personalized 3D face model, andmay refine the shape of the personalized 3D face model so that thedifference in texture patterns may decrease.

The personalized 3D face model acquired through the aforementionedrefining process may be utilized in a variety of application fields, forexample, a facial recognition system and a security/surveillance system.

FIG. 2 is a block diagram illustrating an apparatus 200 for generating apersonalized 3D face model according to example embodiments. Referringto FIG. 2, the personalized 3D face model generating apparatus 200includes a feature point extractor 210, a personalized 3D face modelgenerator 220, and a personalized 3D face model refiner 230.

According to at least some example embodiments, the personalized 3D facemodel generating apparatus 200 receives input images including faceimages of a user and generates a personalized 3D face model representinga shape of the user face. The input images include two or more faceimages. Hereinafter, a description is made for conciseness based on anexample in which the personalized 3D face model generating apparatus 200generates a personalized 3D face model based on a first face image and asecond face image. However, the example embodiments are not limitedthereto and the personalized 3D face model generating apparatus 200 mayalso generate the personalized 3D face model based on three or more faceimages. For example, the personalized 3D face model generating apparatus200 may generate a personalized 3D face model based on a group of imagesincluding a single frontal face image and a plurality of side faceimages.

According to at least one example embodiment of the inventive concepts,the 3D face model generating apparatus 200 may include or be implementedby one or more circuits or circuitry (e.g., hardware) specificallystructured to carry out some or all of the operations described hereinas being performed by the 3D face model generating apparatus 200 (or anelement thereof). According to at least one example embodiment of theinventive concepts, the 3D face model generating apparatus 200 mayinclude or be implemented by a memory and one or more processorsexecuting computer-readable code (e.g., software) that is stored in thememory and includes instructions corresponding to some or all of theoperations described herein as being performed by the 3D face modelgenerating apparatus 200 (or an element thereof). According to at leastone example embodiment of the inventive concepts, the 3D face modelgenerating apparatus 200 may be implemented by, for example, acombination of the above-referenced hardware and processors executingcomputer-readable code.

According to at least some example embodiments, the feature pointextractor 210 extracts facial feature points from input images includingthe first face image and the second face image. For example, the firstface image may be a frontal face image captured from a front of a userface and the second face image may be a side face image captured from aside of the user face. The feature point extractor 210 may detect afacial area of the user by performing face tracking on the input imagesand may extract the facial feature points within the detected facialarea.

Locations of feature points extracted from each of the face images maybe defined in advance, for example, by a manufacture or user of the 3Dface model generating apparatus 200. For example, the feature pointextractor 210 may extract feature points located at eyebrows, eyes, anose, lips, chin, and/or face contour from input images. Thepersonalized 3D face model generating apparatus 200 may extract featurepoints from the input images using a variety of feature point extractionmethods known in the art. For example, the feature point extractor 210may extract facial feature points from input images using an activecontour model (ACM), an active shape model (ASM), an active appearancemodel (AAM), a supervised descent method (SDM), and the like.

According to at least some example embodiments, the personalized 3D facemodel generator 220 deforms a generic 3D face model to a personalized 3Dface model. The personalized 3D face model generating apparatus 200 maygenerate the personalized 3D face model by adjusting a pose and a shapeof the generic 3D face model based on feature points extracted frominput images. The generic 3D face model refers to a deformable 3D shapeface model generated based on training data. The generic 3D face modelincludes vertices each having a spatial location. Spatial locations ofthe vertices may be determined based on a shape control parameterapplied to the generic 3D face model. The personalized 3D face modelgenerator 220 may acquire a 3D shape model by extracting 3D coordinatesabout a shape of the user face based on feature points extracted frominput images, and by deforming the generic 3D face model to be matchedto the extracted 3D coordinates.

The personalized 3D face model generator 220 may generate a furtherprecise personalized 3D face model using feature points extracted fromface images taken at different views. For example, when generating apersonalized 3D face model based on only feature points extracted from afrontal face image, it may be difficult to accurately determine a heightof nose and a shape of cheekbone. Feature points extracted from a faceimage taken at a different view, such as a side face image, includedetailed information about the height of nose and the shape ofcheekbone. Thus, it is possible to further precisely model a 3D facemodel of the user based on the extracted feature points.

The generic 3D face model may use, for example, one or more of a Candideface model, a Warter's face model, and a directly designed face model.The personalized 3D face model generator 220 may generate thepersonalized 3D face model by selecting one of generic 3D face modelsstored in a generic 3D face model database 250 and by matching a shapeof the selected generic 3D face model to feature points extracted frominput images.

For example, the personalized 3D face model generator 220 may deform ageneric 3D face model to a personalized 3D face model by extracting aplurality of landmark points from the generic 3D face model and bymatching feature points extracted from input images to the plurality oflandmark points extracted from the generic 3D face model.

As another example, the personalized 3D face model generator 220 mayacquire a personalized 3D face model by determining shape controlparameters for deforming a shape of a generic 3D face model based onlocations of feature points extracted from input images and by applyingthe determined shape control parameters to the generic 3D face model.For example, the personalized 3D face model generator 220 may determineshape control parameters to be applied to a generic 3D face modelaccording to the following Equation 1:

$\begin{matrix}\begin{matrix}{\tau = {\underset{\tau = {({\tau_{1},\ldots\mspace{14mu},\tau_{n}})}}{\arg\;\min}{\sum\limits_{k = 1}^{m}{{Y_{{2\; D},k} - {P_{k}\left( {\overset{\_}{S} + {\sum{\tau_{i}V_{i}}}} \right)}}}^{2}}}} \\{= {{\underset{\tau = {({\tau_{1},\ldots\mspace{14mu},\tau_{n}})}}{\arg\;\min}\tau^{T}V^{T}{\sum\limits_{k = 1}^{m}{\left( {P^{T}P} \right)V\;\tau}}} - {2{\sum\limits_{k = 1}^{m}{\left( {Y_{{2\; D},k}^{T}P_{k}} \right)V\;\tau}}}}}\end{matrix} & \left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack\end{matrix}$

In Equation 1, Y_(2D,k) denotes locations of feature points extractedfrom a k^(th) face image, m denotes the number of face images, and P_(k)denotes a matrix that performs a 2D perspective projection by warpingthe 3D generic face model based on a face pose estimated from the k^(th)face image. S denotes parameters that represent a mean shape of thegeneric 3D face model, τ has an eigenvalue as shape control parametersto determine a deformation level of a shape of the generic 3D facemodel, and V has an eigenvector value as shape control parameters todetermine a deformation direction of the shape of the generic 3D facemodel. The deformation direction of the shape of the generic 3D facemodel is determined. Further, n denotes the number of vertices thatconstitute the generic 3D face model. In τ_(i) and V_(i), the subscripti denotes an index for identifying a vertex of the shape of the generic3D face model to which each of τ and V is applied.

According to Equation 1, shape control parameters for matching verticesof the generic 3D face model to feature points extracted from faceimages by adjusting spatial locations of vertices in the mean shape ofthe generic 3D face model may be determined. The personalized 3D facemodel generator 220 may generate a 3D face shape model that represents ashape of the user face by applying the determined shape controlparameters to vertices of the generic 3D face model.

The personalized 3D face model generator 220 may select a singlereference face image from among input images and may generate a 3D facetexture model based on texture information of the reference face image.The 3D face shape model refers to a model that includes shapeinformation of the user face, and the 3D face texture model refers to amodel that includes texture information in addition to the shapeinformation of the user face. The 3D face shape model and the 3D facetexture model are each types of the personalized 3D face model.

For example, the personalized 3D face model generator 220 may determine,as a reference face image, a face image having a least facial occlusionor a face image that represents a pose closest to a frontal face imageamong input images. The personalized 3D face model generator 220 maygenerate a 3D face texture model by mapping a texture of a referenceface image to the 3D face shape model.

The personalized 3D face model refiner 230 may refine a personalized 3Dface model by comparing texture patterns of face images. Thepersonalized 3D face model refiner 230 may compare texture patterns offace images using the personalized 3D face model, and may refine a shapeof the personalized 3D face model to minimize or, alternatively, reducea difference in texture patterns.

The personalized 3D face model refiner 230 may project the personalized3D face model to each of the first face image and the second face image.The personalized 3D face model refiner 230 may warp the personalized 3Dface model to be matched to a face pose represented in each of faceimages, and may project the warped personalized 3D face model to theface images. The personalized 3D face model refiner 230 may refine thepersonalized 3D face model based on a difference in texture patternsbetween the first face image to which the personalized 3D face model isprojected and the second face image to which the personalized 3D facemodel is projected.

The personalized 3D face model refiner 230 may extract a correspondencepoint between the first face image to which the personalized 3D facemodel is projected and the second face image to which the personalized3D face model is projected. The personalized 3D face model refiner 230may compare a texture pattern of the first face image and a texturepattern of the second face image in a peripheral area of thecorrespondence point, and may refine the shape of the personalized 3Dface model based on a result of comparing the texture patterns. Thepersonalized 3D face model refiner 230 may compare texture patterns ofperipheral areas for the respective correspondence points, and mayadjust spatial locations of vertices constituting the personalized 3Dface model, so that the difference in texture patterns is minimize or,alternatively, reduced.

When projecting the personalized 3D face model to the first face image,the personalized 3D face model may be warped to be matched to a facepose of the first face image and vertices of the personalized 3D facemodel may be projected to the first face image. Similarly, whenprojecting the personalized 3D face model to the second face image, thepersonalized 3D face model may be warped to be matched to a face pose ofthe second face image and vertices of the personalized 3D face model maybe projected to the second face image.

Since an identical set of vertices of the personalized 3D face model isprojected to each of face images, locations of correspondence pointsbetween the face images may be identified based on locations of thevertices of the personalized 3D face model projected to each of the faceimages. According to example embodiments, the personalized 3D face modelrefiner 230 may identify a location of a correspondence point betweenface images to which the personalized 3D face model is projected, basedon a barycentric coordinate characteristic that a barycenter ismaintained irrespective of a deformation of a triangle.

The personalized 3D face model refiner 230 may compare the texturepattern of the first face image and the texture pattern of the secondface image in the peripheral area of the identified correspondencepoint, and may determine whether to adjust a location of a vertex of thepersonalized 3D face model corresponding to the correspondence point.The personalized 3D face model refiner 230 may adjust spatial locationsof vertices of the personalized 3D face model to make the texturepattern of the first face image become similar to (e.g., above athreshold degree of similarity with respect to) the texture pattern ofthe second face image in the peripheral area of the correspondencepoint. The shape of the personalized 3D face model may be furtherprecisely refined by adjusting the spatial locations of vertices.

The personalized 3D face model refiner 230 may determine a posedeformation parameter and a shape control parameter that minimize or,alternatively, reduce a difference in texture patterns between the firstface image and the second face image in the peripheral area of thecorrespondence point. For example, the personalized 3D face modelrefiner 230 may determine a pose deformation parameter and a shapecontrol parameter to refine a personalized 3D face model according tothe following Equation 2. Here, it is assumed that the first face imageis determined as the reference face image.

$\begin{matrix}{\left( {R,\tau} \right)\underset{({R_{i},\tau_{p}})}{\arg\;\min}{\sum\limits_{i}{{I_{ref} - {W\left( {I_{i},{{PR}_{i}\left( {{\overset{\_}{S}}_{p} + {V_{p}\tau_{p}}} \right)}} \right)}}}^{2}}} & \left\lbrack {{Equation}\mspace{14mu} 2} \right\rbrack\end{matrix}$

In Equation 2, S _(p) denotes parameters that represent a mean shape ofthe personalized 3D face model, τ_(p) has an eigenvalue as shape controlparameters to determine a deformation level of a shape of thepersonalized 3D face model. The shape control parameters are used todetermine a spatial location of each of vertices constituting thepersonalized 3D face model. V_(p) has an eigenvector value as shapecontrol parameters to determine a deformation direction of the shape ofthe personalized 3D face model.

The element Ri corresponding to one of pose deformation parametersdenotes a matrix to deform a pose of the personalized 3D face modelbetween the second face image and the first face image. P denotes amatrix that performs a 2D perspective projection of the personalized 3Dface model to which the shape control parameter and the pose deformationparameter are applied. W denotes a warping function that realigns alocation of a texture using shape and location deformation matrices. Iidenotes a texture of the second face image and Iref denotes a texture ofthe first face image determined as the reference face image.

According to Equation 2, pose deformation parameters and shape controlparameters that minimize or, alternatively, reduce a difference intexture patterns between the first face image and the second face imagemay be determined. The personalized 3D face model refiner 230 mayfurther precisely refine the personalized 3D face model by applying thedetermined pose deformation parameters and shape control parameters tothe personalized 3D face model. The personalized 3D face model refiner230 may iteratively refine a shape of the personalized 3D face modeluntil the difference in texture patterns between the first face imageand the second face image satisfies a predetermined or, alternatively,desired condition.

According to example embodiments, the personalized 3D face modelgenerating apparatus 200 may further include a personalized 3D facemodel storage 240. The personalized 3D face model storage 240 may storea finally refined personalized 3D face model. The personalized 3D facemodel storage 240 may store personalized 3D face models corresponding tousers, respectively.

The personalized 3D face model generating apparatus 200 may update apersonalized 3D face model stored in the personalized 3D face modelstorage 240. When a new face image captured from the user face is input,the personalized 3D face model generating apparatus 200 may extractfeature points of a face from the new face image and may update apreviously generated personalized 3D face model based on the extractedfeature points.

FIG. 3 illustrates face images 310, 320, and 330 to describe a processof extracting feature points from input images according to exampleembodiments.

Referring to FIG. 3, the face image 320 denotes a frontal face imagecaptured from a front of a user face, the face image 310 denotes aleft-side face image captured from a left-side of the user face, and theface image 330 denotes a right-side face image captured from aright-side of the user face.

The global 2D shape information and texture information about the userface may be extracted from the face image 320. Further detailedinformation about a shape of the user face may be extracted from theface images 310 and 330.

A personalized 3D face model generating apparatus may extract facialfeature points 315, 325, and 335 from the face images 310, 320, and 330,respectively. The personalized 3D face model generating apparatus maydetect a facial area from each of the face images 310, 320, and 330, andmay extract facial feature points corresponding to eyebrows, eyes, anose, lips, chin, and/or face contour within the detected facial area.For example, the personalized 3D face model generating apparatus mayextract facial feature points from each of the face images 310, 320, and330 using a variety of methods known in the art, for example, an ASM, anAAM, and an SDM

FIG. 4 illustrates face models to describe a process of generating apersonalized 3D face model by deforming a generic 3D face modelaccording to example embodiments.

Referring to FIG. 4, a generic 3D face model 410 and personalized 3Dface models 420 and 430 are illustrated. The generic 3D face model 410refers to a deformable 3D shape model generated based on 3D facetraining data, and also refers to a parametric model capable ofrepresenting a shape of a user face using a mean shape and a parameter.

A personalized 3D face model generating apparatus may personalize thegeneric 3D face model 410 based on a plurality of face images capturedfrom the user face. The personalized 3D face model generating apparatusmay extract facial feature points from the face images, and maydetermine a parameter capable of matching the extracted facial featurepoints to landmark points of the generic 3D face model 410. Thepersonalized 3D face model generating apparatus may generate thepersonalized 3D face models 420 and 430 by applying the determinedparameter to the generic 3D face model 410. Each of the personalized 3Dface models 420 and 430 refers to a 3D face model of which a pose or anexpression is deformable.

The personalized 3D face model 420 refers to a 3D face model thatincludes only shape information and does not include textureinformation. The personalized 3D face model 430 refers to a 3D facemodel that includes both shape information and texture information. Thepersonalized 3D face model generating apparatus may generate thepersonalized 3D face model 430 to which a texture is applied by mappingthe texture extracted from at least one of the face images to thepersonalized 3D face model 420. The personalized 3D face model 430 thatfurther includes texture information has a higher precision than thepersonalized 3D face model 420 that includes only the shape information.In addition, the personalized 3D face model 430 includes a more numberof vertices.

FIG. 5 illustrates face images 510, 520, and 530 to describe a processof projecting a personalized 3D face model to input images, comparingtexture patterns of the input images, and refining the personalized 3Dface model according to example embodiments.

Referring to FIG. 5, the face image 520 refers to a frontal face image,the face image 510 refers to a left-side face image, and the face image530 refers to a right-side face image. Here, it is assumed that the faceimage 520 is determined as a reference face image.

A personalized 3D face model generating apparatus may project apersonalized 3D face model 540 to each of the face images 510, 520, and530. The personalized 3D face model 540 may represent a user face in a3D shape using vertices. The personalized 3D face model generatingapparatus may warp the personalized 3D face model 540 based on a facepose represented in each of the face images 510, 520, and 530, and thenmay perform 2D perspective projection on each of the face images 510,520, and 530.

The personalized 3D face model generating apparatus may extract acorrespondence point among the face images 510, 520, and 530 to whichthe personalized 3D face model 540 is projected. For example, a point570 of the face image 510, a point 550 of the face image 520, and apoint 560 of the face point 530 are assumed as a correspondence point.An area in which the same vertex of the personalized 3D face model 540is located in each of the face images 510, 520, and 530 may bedetermined as an area in which the correspondence point is located.

The personalized 3D face model generating apparatus may compare texturepatterns in a peripheral area of a correspondence point between the faceimage 520 determined as the reference face image and the face image 510,and may refine a shape of the personalized 3D face model 540 based on adifference in the texture patterns. The personalized 3D face modelgenerating apparatus may refine the shape of the personalized 3D facemodel 540 to minimize or, alternatively, reduce the difference intexture patterns between the face image 520 and the face image 510 inthe peripheral area of the correspondence point. For example, accordingto Equation 2, the personalized 3D face model generating apparatus maymore precisely refine the personalized 3D face model 540 by determininga pose deformation parameter and a shape control parameter to minimizeor, alternatively, reduce a difference in texture patterns between theface images 510 and 520 in a peripheral area of each correspondencepoint, and by applying the determined pose deformation parameter andshape control parameter to the personalized 3D face model 540.

The personalized 3D face model generating apparatus may compare texturepatterns only in a peripheral area of a correspondence point between theface image 520 determined as the reference face image and the face image530, and may refine a shape of the personalized 3D face model 540 tominimize or, alternatively, reduce a difference in the texture patternsin the peripheral area of the correspondence point.

FIG. 6 is a flowchart illustrating a method of generating a personalized3D face model according to example embodiments.

In operation 610, a personalized 3D face model generating apparatusextracts facial feature points from input images. The personalized 3Dface model generating apparatus may detect a facial area of a user byperforming face tracking on the input images and may extract the facialfeature points within the detected facial area. For example, thepersonalized 3D face model generating apparatus may extract facialfeature points located at eyebrows, eyes, a nose, lips, chin, and/orface contour from input images using an ACM, an ASM, an AAM, an SDM, andthe like.

In operation 620, the personalized 3D face model generating apparatusmay deform a generic 3D face model to the personalized 3D face modelbased on the extracted feature points. The personalized 3D face modelgenerating apparatus may acquire a 3D shape model by extracting 3Dcoordinates based on the feature points extracted from the input imagesand by deforming the generic 3D face model to be matched to theextracted 3D coordinates.

The generic 3D face model may use, for example, a Candide face model, aWarter's face model, and a directly designed face model. Thepersonalized 3D face model generating apparatus may generate thepersonalized 3D face model by selecting one of generic 3D face modelsand by matching a shape of the selected generic 3D face model to featurepoints extracted from input images. The personalized 3D face modelgenerating apparatus may acquire a personalized 3D face model bydetermining shape control parameters for deforming a shape of thegeneric 3D face model based on locations of feature points extractedfrom input images and by applying the determined shape controlparameters to the generic 3D face model.

In operation 630, the personalized 3D face model generating apparatusmay project the personalized 3D face model to each of the input images.The personalized 3D face model generating apparatus may warp thepersonalized 3D face model to be matched to a face pose represented ineach of face images, and may project the warped personalized 3D facemodel to the face images.

In operation 640, the personalized 3D face model generating apparatusmay compare texture patterns of face images based on the personalized 3Dface model and may refine the personalized 3D face model based on aresult of comparing the texture patterns. For example, the personalized3D face model generating apparatus may refine a personalized 3D facemodel based on a difference in texture patterns between a first faceimage to which the personalized 3D face model is projected and a secondface image to which the personalized 3D face model is projected. Thepersonalized 3D face model generating apparatus may refine a shape ofthe personalized 3D face model to minimize or, alternatively, reduce thedifference in texture patterns between face images based on thepersonalized 3D face model. Operation 640 will be further described withreference to FIG. 7.

FIG. 7 is a flowchart illustrating in detail operation 640 of refining apersonalized 3D face model in a method of generating the personalized 3Dface model according to example embodiments.

In operation 710, the personalized 3D face model generating apparatusmay extract a correspondence point between the first face image to whichthe personalized 3D face model is projected and the second face image towhich the personalized 3D face model is projected. The personalized 3Dface model generating apparatus may extract a correspondence pointbetween face images based on locations of vertices of the personalized3D face model projected to each of the face images.

In operation 720, the personalized 3D face model generating apparatusmay refine the shape of the personalized 3D face model to make thetexture pattern of the first face image and the texture pattern of thesecond face image become similar to (e.g., above a threshold degree ofsimilarity with respect to) each other in the peripheral area of thecorrespondence point. The personalized 3D face model generatingapparatus may refine the shape of the personalized 3D face model tominimize or, alternatively, reduce the difference in texture patternsbetween face images in the peripheral area of the correspondence point.For example, according to Equation 2, the personalized 3D face modelgenerating apparatus may refine a shape of a personalized 3D face modelby determining a pose deformation parameter and a shape controlparameter that minimize or, alternatively, reduce a difference intexture patterns between face images and by applying the determined posedeformation parameter and shape control parameter to the personalized 3Dface model.

The units and/or modules described herein may be implemented usinghardware components and software components. For example, the hardwarecomponents may include microphones, amplifiers, band-pass filters, audioto digital convertors, and processing devices. A processing device maybe implemented using one or more hardware device configured to carry outand/or execute program code by performing arithmetical, logical, andinput/output operations. The processing device(s) may include aprocessor, a controller and an arithmetic logic unit, a digital signalprocessor, a microcomputer, a field programmable array, a programmablelogic unit, a microprocessor or any other device capable of respondingto and executing instructions in a defined manner. The processing devicemay run an operating system (OS) and one or more software applicationsthat run on the OS. The processing device also may access, store,manipulate, process, and create data in response to execution of thesoftware. For purpose of simplicity, the description of a processingdevice is used as singular; however, one skilled in the art willappreciated that a processing device may include multiple processingelements and multiple types of processing elements. For example, aprocessing device may include multiple processors or a processor and acontroller. In addition, different processing configurations arepossible, such a parallel processors.

The software may include a computer program, a piece of code, aninstruction, or some combination thereof, to independently orcollectively instruct and/or configure the processing device to operateas desired, thereby deforming the processing device into a specialpurpose processor. Software and data may be embodied permanently ortemporarily in any type of machine, component, physical or virtualequipment, computer storage medium or device, or in a propagated signalwave capable of providing instructions or data to or being interpretedby the processing device. The software also may be distributed overnetwork coupled computer systems so that the software is stored andexecuted in a distributed fashion. The software and data may be storedby one or more non-transitory computer readable recording mediums.

The methods according to the above-described example embodiments may berecorded in non-transitory computer-readable media including programinstructions to implement various operations of the above-describedexample embodiments. The media may also include, alone or in combinationwith the program instructions, data files, data structures, and thelike. The program instructions recorded on the media may be thosespecially designed and constructed for the purposes of exampleembodiments, or they may be of the kind well-known and available tothose having skill in the computer software arts. Examples ofnon-transitory computer-readable media include magnetic media such ashard disks, floppy disks, and magnetic tape; optical media such asCD-ROM discs, DVDs, and/or Blue-ray discs; magneto-optical media such asoptical discs; and hardware devices that are specially configured tostore and perform program instructions, such as read-only memory (ROM),random access memory (RAM), flash memory (e.g., USB flash drives, memorycards, memory sticks, etc.), and the like. Examples of programinstructions include both machine code, such as produced by a compiler,and files containing higher level code that may be executed by thecomputer using an interpreter. The above-described devices may beconfigured to act as one or more software modules in order to performthe operations of the above-described example embodiments, or viceversa.

Example embodiments of the inventive concepts having thus beendescribed, it will be obvious that the same may be varied in many ways.Such variations are not to be regarded as a departure from the intendedspirit and scope of example embodiments of the inventive concepts, andall such modifications as would be obvious to one skilled in the art areintended to be included within the scope of the following claims.

What is claimed is:
 1. A method of generating a three-dimensional (3D)face model, the method comprising: projecting a personalized 3D facemodel to each of images associated with a user face; and refining thepersonalized 3D face model based on the images to which the personalized3D face model is projected, wherein the refining comprises: extracting acorrespondence point between the images to which the personalized 3Dface model is projected; and refining a shape of the personalized 3Dface model such that a similarity between texture patterns of the imagesto which the personalized 3D face model is projected increases in aperipheral area of the correspondence point.
 2. The method of claim 1,wherein the refining comprises: generating a comparison between thetexture patterns of the images to which the personalized 3D face modelis projected in the peripheral area of the correspondence point; andrefining the shape of the personalized 3D face model based on thecomparison.
 3. The method of claim 2, wherein the refining includes,iteratively, determining if a first condition is satisfied, and refiningthe shape of the personalized 3D face model based on the comparisonbetween the texture patterns, until the first condition is satisfied. 4.The method of claim 1, wherein the refining comprises: refining theshape of the personalized 3D face model such that the similarity betweenthe texture patterns of the images to which the personalized 3D facemodel is projected exceeds a threshold degree of similarity in theperipheral area of the correspondence point.
 5. The method of claim 1,wherein the refining comprises: determining a pose deformation parameterand a shape control parameter that reduce a difference between thetexture patterns of the images to which the personalized 3D face modelis projected in the peripheral area of the correspondence point; andapplying the determined pose deformation parameter and shape controlparameter to the personalized 3D face model.
 6. The method of claim 5,wherein the shape control parameter controls a spatial location of eachof vertices constituting the personalized 3D face model.
 7. The methodof claim 1, wherein the projecting comprises: projecting thepersonalized 3D face model to each of the images based on a facial poseincluded in each of the images.
 8. A non-transitory computer-readablemedium storing computer-executable instructions that, when executed byone or more processors, cause the one or more processors to performoperations including, projecting a personalized 3D face model to each ofimages associated with a user face; and refining the personalized 3Dface model based on the images to which the personalized 3D face modelis projected, wherein the refining comprises: extracting acorrespondence point between the images to which the personalized 3Dface model is projected, and refining a shape of the personalized 3Dface model such that a similarity between texture patterns of the imagesto which the personalized 3D face model is projected increases in aperipheral area of the correspondence point.
 9. An apparatus forgenerating a three-dimensional (3D) face model, the apparatuscomprising: a memory storing computer-executable instructions; and oneor more processors configured to execute the computer-executableinstructions such that the one or more processors are configured to,project a personalized 3D face model to each of images associated with auser face; extract a correspondence point between the images to whichthe personalized 3D face model is projected; and refine a shape of thepersonalized 3D face model such that a similarity between texturepatterns of the images to which the personalized 3D face model isprojected increases in a peripheral area of the correspondence point.10. The apparatus of claim 9, wherein the one or more processors arefurther configured to execute the computer-executable instructions suchthat the one or more processors are further configured to, generate acomparison between the texture patterns of the images to which thepersonalized 3D face model is projected in the peripheral area of thecorrespondence point; and refine the shape of the personalized 3D facemodel based on the comparison.
 11. The apparatus of claim 10, whereinthe one or more processors are further configured to execute thecomputer-executable instructions such that the one or more processorsare further configured to, iteratively, determine if a first conditionis satisfied, and refine the shape of the personalized 3D face modelbased on the comparison between the texture patterns, until the firstcondition is satisfied.
 12. The apparatus of claim 9, wherein the one ormore processors are further configured to execute thecomputer-executable instructions such that the one or more processorsare further configured to refine the shape of the personalized 3D facemodel such that the similarity between the texture patterns of theimages to which the personalized 3D face model is projected exceeds athreshold degree of similarity in the peripheral area of thecorrespondence point.
 13. The apparatus of claim 9, wherein the one ormore processors are further configured to execute thecomputer-executable instructions such that the one or more processorsare further configured to, determine a pose deformation parameter and ashape control parameter that reduce a difference between the texturepatterns of the images to which the personalized 3D face model isprojected in the peripheral area of the correspondence point; and applythe determined pose deformation parameter and shape control parameter tothe personalized 3D face model.
 14. The apparatus of claim 10, whereinthe one or more processors are further configured to execute thecomputer-executable instructions such that the one or more processorsare further configured to store the refined personalized 3D face model.